Temporal spatio-velocity transform and its application to tracking and interaction

This paper describes the temporal spatio-velocity (TSV) transform for extracting pixel velocities from binary image sequences. The TSV transform is derived from the Hough transform over windowed spatio-temporal images. We present the methodology of the transform and its implementation in an iterative computational form. The intensity at each pixel in the TSV image represents a measure of the likelihood of occurrence of a pixel with instantaneous velocity in the current position. Binarization of the TSV image extracts blobs based on the similarity of velocity and position. The TSV transform provides an efficient way to remove noise by focusing on stable velocities, and constructs noise-free blobs. We apply the transform to tracking human figures in a sidewalk environment and extend its use to an interaction recognition system. The system performs background subtraction to separate the foreground image from the background, extracts standing human objects and generates a one-dimensional binary image sequence. The TSV transform takes the one-dimensional image sequence and yields the TSV images. Thresholding of the TSV image generates the human blobs. We obtain the human trajectories by associating the segmented blobs over time using blob features. We analyze the motion-state transitions of human interactions, which we consider to be combinations of ten simple interaction units (SIUs). Our system recognizes the 10 SIUs by analyzing the shape of the human trajectory. We illustrate the TSV transform and its application to real images for human segmentation, tracking and interaction classification. 2004 Elsevier Inc. All rights reserved. 1077-3142/$ see front matter 2004 Elsevier Inc. All rights reserved. doi:10.1016/j.cviu.2004.02.003 * Corresponding author. Fax: 1-512-471-5532. E-mail addresses: [email protected] (K. Sato), [email protected] (J.K. Aggarwal). K. Sato, J.K. Aggarwal / Computer Vision and Image Understanding 96 (2004) 100–128 101